cat. / + DOCORENT RESUME 


. EC 123 909 


ED 185. 758 7 . 
AUTHOR Mirki@, Phvllis K.: Deno, Stanley L. Pi 
TITLE ‘Formative Evaluation in the Geaeen cen An REpEOSCh to . 
. , Improving Tnetruction. | : 
INSTITUTION Minnesota Univ., Minneapolis. Tet. for Research on 
Learning Disabilities. . 
-SPONS os Bureau of Education for the Handicapped (DHEW/OE), | 
Washington, D.C. «a- 
FEPORT "NO IRLD-RR-10 , 
* pUB DATE . <« Auq’79 ; : 
CONTRACT -300-77-0491. 
NOTE ¥ 35p.: See also EC 123 901-925. ae, 
# “ee 
- EDRS PRICE — MFO1/PC02 Plus Postaqe. 
DESCRIPTORS * - Elementary Fducation: FvaPuation Methcds;, Exceptional v 
: —--Child Research:: *Formative Evaluation: *Learning % 
Disabilities: *Mbld Mental Petardation: *Student . 
Eyaluation ‘ 
ABSTRACT | 


- The éffects of two components of formative evaluation ° 
(frequency of measurement and data utilization). were compared to 
isolate Formative evaluation components which teachers might oa 
routinely use, to monitor achievement in learning disabled stident 
Fifty-two learning disabled and educatle mentally retarded elementary 
students enrolled in regular class praqrams and receiving reading 

“instruction in a special education resource rocm were randomly 
assigned to either a. pretest nen- data-based change group, a daily 
measurement-non-data-based chanae group, a daily ieasurement 
data-based group, or at untteated control qroup. Analysis of results 
of oral reading data supported daily measurement and datazbased 
ee: as effective compchents-of formative evaluation. ‘Mut hor) 


° 


* Reproductions supplied by EDRS*are the best that'can be made\ 


* * frem tha original document. 
NOI CIMIO IO IOI IOI I IOI IOI IOII IOI I OIIDIGI SII I OIGISIOI I IGIOI II IOI ISI IOI IORI foiok ak III I i 


ERIC . os oo | a 


- i or ae : a : ¢ US. DEPARTMENT OF 4 
be . : . : EALTH, 
L oa b ‘oe : a _ + EOUCATIONA WELFARE 
2 a ; NATIONAL INSTITUTE OF 
. ae EDUCATION 


THIS A gpnead HAS 

; BEEN REPRO- 
sah EXACTLY as RECEIVED EM ’ 
HE PERSONOR ORGANI . 


=f 
x. 

ATING IT POINT : 
uv my) University of Minnesota piranha tte 

: CIAL aoe INSTITUTE OF 
ssa ; . a ay 4 ‘ 

. . ‘ 
' . 


¥ : 
“ | : \ 
* Research Report No.* 10 7 
: ; i ‘ - , a ; . Y 
~~ FORMATIVE EVALUATION ‘IN THE CLASSROOM: “ 
\ » oc , AN APPROACH TO IMPROVING INSTRUCTION — | 
| Phyllis,K. Mirkin and Stanley L. Deno- P 


Institute for. 
- Researchon 
‘Learning «+ | 
. Disabilities ot 


+4} MISSION TO REPRODUGE THIS a 
MATERIAL HAS BEEN GHANTED BY | ; 
a . . 
~ James Ysseldyke er 
\ . 
" 10 rte LOU TONAL RESOURCES — - . . , 


a 7 < ‘ RF RIG)” . 
EF} IC ee i | ) 
ate : : . 

‘ ‘ 


Pu 


o 


’ Associate Director: 


The: Institute for Research on Learning Disabilities is supported by 


% 


aa 


° 


ine 


Director: James E. Ysseldyke ‘ 4 


Phyllis-K. Mirkin 


x & 7 , f 


¢ 


a contract (300-77-0491) with the Bureau of Education for the Handi- * 


capped, Department of Heafth, Education, and Welfare, U.S: Office of 
Education, through Title VI-G of Public Law 91-230. 
tigators are conducting research on the assessment /dec ision-making/ 


Institute inves- 


; _* idntervention process as it relates to learning disabled children. ” 
Research activities are organized into eight major areas: - a 
Ly Adequacy of Norm-Referenced Data for Prediction \ 
of Success. 
-~ =: . . ,, ~ = 
Il. Computer Simulation Research on the Assessment/ - 
; Decision+making/ Intervention Process 
III. Comparative Research on Children Labeled’LD and 
j -  Children’Failing Academically bu not Labeled LD 
IV. ‘Surveys..on’ Ti-the-Fiel ‘Assessffertt, Decision Making, 
and Intervention, | _— ) 
3 ye: Ethological. Research on P acement| Team Decision 
' -f Making 4.0 
VI. Bias Following Assessment ‘ 
; ¥ "VAI. Reliability and Validity of Formative Evaluation 
|...” Procedures 7 a \ 
_VITL. Data-Utilizatiipn Systems in‘ Instructional Pro- 
. - gramming : * 4 ; ; 
-- Additional information on these ‘research areas may be obtained by writting 


oe to the Editor at the Institute. 


fay 


J 


™ 


ee nr st eee 


e 


The research reported herein was conducted under .government sponsor- 


ship. . Contractors are encouraged to express freely their professional 
judgment In the conduct of the project. 


Points of view or opinjlons 


‘stated do not, therefore, necessarily represent official position of 
the Bureay of Education for the Handicapped. ( 
E Oo ‘s ; we ° * 


ae te ae eee 


eee 


i a te ee -_ a! 


Research Report No. 10 i, 


r 7 


FORMATIVE EVALUATION IN THE CLASSROOM:: 
"AN APPROACH TQ IMPROVING INSTRUCTION 


\ 


. ¢ . 
oon . 
«2 


“Phyllis K. ‘Mirkin and Stanley L. Deno. 


Institute for Research on Learning Disabilities 


University of Minnesota: "s 


. August 1979 | G . 


~~ = | 


se 


 % 7 me ‘ 
ve Abstract ° : i 
F Be i \ 


” 
4 


+ The effects of, two components of formative evaluation, (a) lay aad P 
‘ : : ¢ i : 


of measurement .afid (b) data utilization, were.compared in order to isolate 


4 : < , 


. . 1 
’ 


formative evaluation‘ components which teachers might routinely use to 

Us “3 s ‘i : 
monitor achievement. Fifty-two learning disabled and educable mentally 
retarded students enrolled in regular class programs and ,receiving reading 


instruction in a special education resource room were randomly assigned 


ay ; 


to either ) a pre-posttest non-data-based*change group, (b) a daily 
measurement’ non-data-based ghange group, (c) a daily measurement data- 


s 


based change group, or (d) an untreatpd control group. Analysis of 


‘results of oral reading data supported daily measurement and data-based 
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‘ . changes as effective components of formative evaluation. . 
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Formative Evalyation in the Classroom: An Approach to Improying Instruction 
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a Recent regulations promulgated under the Education for all Handi- 
ise 1 a : . 

capped Children Act of 1975, PL 94-142, require the development of an | 

‘Individual Educat tonal veeean GTEED which specifies “annual and short- 


term objectives whenever a stident is identified as requiring epecial 
1 


education service. While logical arguments to support use of objectives 


{ 
in-the development of educational programs have been proposed (Mager, 1962; 


Y 


Popham & Husek,’ 1969; Steiner, 1975; Tyler, 1950 becempicical ‘yeri fication 


of ‘the beneficial achievement effects of specifying objectives ts lacking. 
. Equal numbers of studies can he found in which significant and non- signi- 


_ficant results are reported (Duchastel & Merrill, 1973; Hartley & Davies, 


« 
- 


. ada A major factor in these equivoeal results may be the yack of « 


& ’ 
sdeciiabe eva'luation procedures to assist teachers ‘in effective decision 
« A . ’ 
faking during the instructional program (Crutcher & Hofmeister, 1975). 


° *. 


*radi¥tonally: educational ealneton hus been oriented to placement 


e 
° , 


and’ summative decision making. While. psychologists and -educat ional’ diag- 


nosticians routinely use diagnostic testing procedures which have forma-; 


\ ; , 7 
tive decision-making potential, these procedures, are not the usual class- 


.» room practice. Review: and, reteaching are the usual instructional deci- 


‘ 


‘N ae ; 

sioys and relate only to items missed in the post test. A study of ' 
teacher decision making by Zoharik (1975) supports this view. He found 
planning decisions regarding evaluation, diagnosis, and instructional 


strategies were made by fewer than one-third of the 194 teachers studied. 


. 


“Similar-findings were previously reported by Goodlad and Klein (1974) 


and Popham and Baker (1970)... ‘ . . 7 


‘Formative evaluation is concerned with the evajuation of edutdtional 
” see oar “ > * . : ‘ ‘ 
‘programs still in some stage of development (Scriven, 1967)., Unlike 
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placement and summative evaluation, Pornative evaluation is intended to 


A 


lead to the Sap coveneak of SENSE NON: during the teaching process its 
my 
‘ : 
¢ 
seit by providing feedback to both teacher and student. regarding, ob- 
2 


jective mastery (Conroy, 1973; Popham, 1972; Snow, 1977; Sullivan, 1971; 
4 Bt oe 


Sherman, Note 1). . _ * ; 

a “* ee . 
While there is considerable agreement that the key to improved in- 
. > . 


struction and educational, decision making by. teachers may be “formative ~ 


evaluation procedures, the most errpctive components af a formative eval=, 


~ 


( autionesystem have not been dsolated or systematically compared (Sullivan, 


1971). Sullivan recommends identi fication of precise a chm fin initial 
‘ at * 6 IS . , . 
pisnntns and the deve lopment of a detailed system for monitoring and re- 


Q 


‘ =A 
cording achievement’ of objectives: as important to the succéss of a forma- 


tive evaluation system. imporfent concerns remain, however, regarding. 


ce ¥ ’ ‘ . ae *. 
a) the frequency of test administrafion’required to make appropriate de- 
. : . t 

cisions uring the instructional. program, and (h)’ the way in which the 
’ . 7 ; diese wo? 
collected -data are utilized. 
*e@ _ : y . ‘3 . 
* Recommendations regarding frequency of ‘measurement vary from the 


: : : V a 
periodic pre-post measurement approach described by Van. Etten and Van 


Etten (1976) af non-cont inuous measurement, to’ the direct and“daily. 
‘continuous measurement BPRrOgeR advocated by ttese who practice 
a ° 

7” technodogy of preciaron teaching (Alper & White, 1971; Haughton, 1971; 
Kunzelman, 1970; Pindstey 1964; Spowiee 1967; White & Haring, 1976; 
faring’ Lovitt, Note 2, Starlin, Note 3, White & hberty. Note 4). 

. It is argued by proponents of this view that srigecoavinuons measure- + 
ment and analysis ot seeiet anne permits the teachér to aiailee changes 


. 


in the pronnes de it will be of -the greatest benefit to the student 
’ , / 


(Starlin, 1971). - 2 en, 


‘5 a The issie of "data utilization" is also one which has note been — . 


eo 


adequately: resopved!with “respect to formative evaluation. One solution - 


has been to establish a set of rules which provides a standard method _ 2 


for daily program analysis (Liberty, Notes 5 dnd 6). The rules attempt 


to take the "guesswork".out of analysis of daily measurement data, by . 


‘ a 


. : 4 = 


providing guidelines whth respect to the length of time an intervention 
° . should be maintained for {ndividuak programs. The rules are determined, 


not only by the progress of the student. but, by the objective (aim) of | ? 
; J ° ; . . . a 

the program as well. The rules add an‘ important dimension to formative 
evaluation not addressed in the pre-posttest paradigh.~ 


A limited number of studies is reported in. the research literature. 


. . 
F , Ps . «a 


ents of a formative evaluation system. Jenkins, Mayhall, Pesghka, and 


oa 


where attempts have been, made. to systematically isolate effective compqn- 
. : » . \ 


. 


5 7 . : .- ; * 
* ‘Townsend (19-74): compared charted and non-charted feedback of ‘daily measure-. 


Pa 
* 1 


i ° . a : bal - 5 ¢ 
ment joel to teachers and students and reported results which significantly 
favored the charted feedback group. Frumess (Note 7) compared different 


degrees of self-management when used with daily measurement and found 


ae - : : fs 


significant differences favoring students who charted their own daily . 


& ‘ re 


‘ & 


scores compared to students for whom there was no.self-charting or 


t®acher charting of performance., . » 


2 . 
t 


In an investigation of daily measurement and decision ‘rules, 


Bohannon (Note 8) compared teacher judgment ag the predominant, formative io 


evaluation procedure with daily measurement and data decision rules: 
Ps e Ce 
( = . 
he reported results which favored-students in ther latter treatment. . 
Of ,particular-gnterest were findings which suggested that for eight of ( 
: » Ma » ; : 


; ae r ; a : 


is , ; & ; ' 
the 23 students in the study one minute of daily measurement was suf- oan ° 
* . - a a ° 
’ ; ‘ : 
ficient to improve achievement, thus making. it unnecéssary for the - - . 


* 1 


‘teacher to use any decision rules to make program adjustments. 


No 
: The present study further explored Bohannon's findings by con-- 
t : . . J = : : a ; ‘ + 
.trasting student achievement under conditions of daily measurement and. ‘ 
, edaily measurement with data decisfon rules. In addition, a third treat- 
* ment was initiated in which pre-posttest measurement was the only forma~ ~~ 
— ; 2% ; —* ‘> 
tive evaluation procedure systematically implemented by the teacher. 8 
‘ “e + 7 ~ ‘ . > : : , 
+ The research was designed to answer the following questions: =e 
LF 2 ; , . - 7 : 
. : . 7 - ° : . ; 4 
. 1. Does daily measurement increase. student performance on . 
. ~ objectives beyond that attained with pre: and “post testing? y 
- 4 oe 
: 2. Dods adding-a data utilization component increase student . 
: ; performancé dn objectives beyond- that attained with daily 
: . : +, eo i . = e 
: : measurement alone? a » . . . 
. } . ‘ * . 
Method a oe “ee 
Participants . : . . 7 . : 
‘ . m0 ; : . ° i ‘ * 
ye Fifty-two children in.grades two through six who had been previously , 
classified as learning -disabled oO cable mentally retarded participated 


in, the study. The students were enrolled in ‘regular class programs and ‘* 


e t 
, were receiving daily reading instruction -from 13 special, education re- , 


. - . i . 
source teachers in four, metropolitan school districts. in Minnesota. 
aa ; . 


’ 


Treatments : ; 


Four students were randomly selected from each resource teacher's 


existing caseload and randomly assigned to one of. three experinghtal 


t 
* 


treatment groups or to an untreated control group. The 13 subjects: 
. ~ q 
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“post measurement, non-data-based change’ (PPN), (b) daily ineasurement , 


measures of oral reading. correct and dncorreet gad vocabulary meaning 


in each treatment group then received reading instruction which included 


€, 


one of three combinations of formative evaluation procedures: (a) pre- 


,> 
. 


non-data- based change (DMN), or (c) daily measurement , ‘data-based change 
(DMD). ‘Analyses of variance of pretest ‘performances ery eres 


differences between groups. 


Instfuments © . ; . 


. . e oi 
; . @ ; : 
Four types of data were used to analyze treatment effects. Measures 


’ bs. 


of oral reading rate correct, oral reading rate incorrect, vocabulary , 
= 


‘meaning, and Eomee en een were obtained for all students both prior to and 


following treatment. The first three measures were derived fram stories *.. 


randomly selected from Levels Ib IIb, and IlIb of die Pouer “wutider Kits 


™, 


. 
‘ 


(SRA, 1963, 1969). Each student read. orally for three minutes ang was, 


, % 


_asked to define five words which had’ previously been randomly selected 


. ° . 
' . 


from the first 100 words of the story. The tdétal number of words read 


‘correctly and incorrectly were then counted and divided by three to obtain. 


the per minute rate. The total number 5£ words defined correctly was 


determined by teacher judgment. When in doubt, the first definition in 

: : ‘ ‘ x 2 j . ¥ 
ry - 4. 

the ‘dictionary was, used as the criterion. A measure of each student's ° 


oy 
° . , 


xeading comprehension was’ obtained using the Stanford Diagnostic Readitg 


‘Test hayes I, Forms W and X (1968) and the comprehens’ion subtest of the 


Stanford Diagnostic Reading Test,- Level II, Forms W and X (1966). ~ Daily: 


». 
mn s 


in the SRA Power Builder Stories¥ also were obtained for students in fhe 


( 


daily measurement and data decision rule groups (N‘= 26). 4 


¢ . * 


s _ & ‘ 


1 


e 


6 


“Speci is peoceauses 2 ie . 


eG All subjects initially. were placed, cox: reading instruction by 


ne experimenter in three, levels of the SRA Power Builder Kits (1963, ro 
“Placgrent was determined _ identifying passages which the student eee a 


a 


read orally at the rate of 50 to #5, 35 to 60,. and 30 to 40 words correctly 


; 1 - © x ‘ . 
per minute. Error performance was not’ used in making the placement .deci- 


a * : 


sion. ; ; : 
a % 
a * } 


Each student's oral beading performance | was then measured for three 


+ days at each’ of the three. rete to reliably scien tik initial berformance. 


A 30 percent increase in oral reading rate correct was arbitrarily established 


as the 18- day objective eon all stusen, in thef experimental treatments. The 


~ . 


desired level at 18 dye was determined by multiplying the aeiteat initial 
oral reading. correct score at each level by a factor af i:3.. To establish 


daily objectives for the daily.meagurement and data decision rule groups, a 


straight increasing daily aim line was drawn on an equal interval graph : 


. a 
. eenneer te ches median ‘initial level. with the fed) aa level at 18 ve 
“8 . : 
(Liberty, Notes: 5 and) The daily objective for error rate was to re- 
: s é 
, aaiae tee below the median initial error rate. This objective was, shown 
vA A) 
vw 
on the equal interval graph by drawing a straight line across the graph 
- * ‘ ? . _ 
at the ie a Ss feat median error rate for each kevel. An example of 


, 


a graph Te initial data points and daily ain line drawn for both correct 


J ' ° 


° ‘ e 


and grror Fj is shown in Figure l. ia ; 7 * . 


eed 


a n 


The sequence of instructional activities for all groups was as 


e - ry 


follows: 


. 


te - Each student received 20 minutes of reading instruction daily |» 


a 


from the special education resource teacher.- Instruction consisted of ° af 


reading nine stories twice at each level over an 18-day period. -Students .« 
* : +: aoe - 


eo ; a . : 
ad~& loud for three minutes at each of the ,three Placement levels.. 


St dents. Were’ then asked to define five words from each story. ‘Error. 


‘cornection ‘and word meaning correction were given. 


‘ 


\ Each of the treatment groups differed. from one another Avith respect 


° 


to the\ daily forniative..evaluation procedures used as follows: , 2 


e 


a Daily Measurement, I Data-based Change _{DMD) The seacnem and 


° a 


student reviewed the graph each day to deternine whether the daily 
, objective had been acnieued. If daily data points were plotted beled 


* the aim line for two consecutive data days. a new aii ‘line wag drawn - 
be . ’ a : 
parallel to the original line (i.e., the target date ‘was. extended)’ and: a 


on. 


a program change was made. . If the 2’ data points- were plotted above 
util 


\ : = % 
the daily. aim ‘line for five consec days, a new daily aim Lind was 
drawn, parallel and abote the original Tine and a -program ‘change was} 


. 
. 


. 


‘= 
tiade. Examples of original and redrawn daily aim lines are shown.in 


” 


= Figure 22 ) ' : ‘ 


-? 7 Se ee ee ee ee ee 


ld ca ‘Insert Figure 2 about here 


4 C ° . ig ws t Pe; ° ‘ 
Ss Error data were also reviewed daily. If daily data’ points were 
Pa < ‘ ? . * t . 

: a Mae : fs : ; , \ : . 
’ ‘plotted above the median error line for two data days; a new median” 
error: line was drawn and a program change wal made. If error data _ > 
yo we're plotted below the median line ioe five days, thé same Procedure | 

e \ , ern ee 
was followed. 


. ” a ° ‘ 7 : : . 
re © 19 3 Fi . « ' 
a a are _ ; , foe 
*? . : . = : . 
By roe : 
aed x Lae e . . 
= a4 . Me ‘ . 


z z¢ vo tne ee 2 4 2 . . 
Lt . . The'’teachers made, a series of: program changes as.a function 
: - * «© Ee 


ae ; : - pte e ; 
“a. | OF the student's vere . The -changes were, in sequence: . . 

. ° . qt ; ty 
. + o : ° b e . ~ . e - 


Co ee Each ‘day a data point was plotted on: or sabove the lihe ah 


‘ vite Student, ‘received. eG gum ball dispensed, by niabink-3 me ae 


of . ” Moy, diet p ate de 


y ayboed by the teacher in a- “gem al achines ‘i 8 a 


~~ 


so. 2. 4 a 2. ' Bach day a date paint er eer on or above the line _ ; 4 


3 -_, @ . 
, nn a : ie - 4 _ \’ ° 
a ee ae oe the > studeng, réceived, a ee of. his/her choice. ' : a 
. ? = * . a @ ~ LY 2 
3.  Eath day a data point ‘was plotted én oF above? the tine e 
. : e Le a 9, Pa os 


e . pe é : . 


et ‘ ‘card. ' Five «opts coul tbe \ rehthged for a igpatbic item ; 


: such as‘a Gate folder, “in opportdnitys to wom ee the offices oe 
. ae ee an ia _! an , o 2° 


eA or operate the audiovisual, eqajfment, | ér - ether, etetlar ‘ 


as a adh : t e a i 4 Pi 
. ‘ sthool activity waged: on individual interest, oy ” = 
« “a Daily Measurement N None -data-based | Change DN) Foldowing tined.« _ ‘ 

i we aval reading, ‘teacher and étudent marked. ite graph and shecieed “6 ‘see . ~ <. \ 
a a ve: ‘ ‘ 
. - wheviier tite daily Spiective had bee achieved: ‘Teachers provided encourage- ve 
fe ee with positive ‘statements and praise: ‘Whénever progranr chatiges Ore 


+8 a inplenented for’ the (DMD) erours eyes: also implemented fae this’. 
% aa e 2 ae me > . F : we = * 


bf * : « 
» Beoup. 7 aon? . ce. | | — 
aden se . ° ° 


id rs Pre-post Measurement Non-data~based_ Ch Change _(PPN) : Following “daily se 


‘ey i 
$ ON a 
‘~ oJ * 


“4 6ral- reading” teachers praised eeudents and thanked them for reading. 
a ad . on 


- ‘Whenever program changes were implemented for the" DMN group, they were 


. 
or 


= ee mae implenented for: ‘this goa ao . a . | . o 


7 : « 


“ . ‘om Untreated Control Grou up (ue): This ereuy: game We “the resource ‘s 


“ e room daily for’ ‘regular reading dettuerion ‘of approximately the same 


t a . 


®, _ 
Ju neteo weds ds the experimental groups. ‘No controls were 


. ae 


“eo 
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v 


t ae ‘ , 
; - exerted over the. reading instruction of students in this group. ‘ 


/ _ Immediately following the 18-day instruction period; the oral 


meaning performance of all students was 


- 


. sce 8 : ; 
at each. of one three levels -in which initial ’ 


reading fluency and vocabula 


again measured on three day 


. 


- per formance was Naina’. On indeenivd days, a measure of reading com- 


“ prehension \ was also obtained using the Stanford Diagnostic: Reading. Test 


Leve 1 VU Form W. (1968) for students in geadee two and three and the Stdn- “= & 


vos 


i. Yordbiagnostic Reading Test, Level II. Porn W (1968) for students th * : a . 


- ae +e 
> 


‘grades four through six. 


: » Results cL ; 
eee “4 ° ae é a ° 


The post treatment data for all groups on all. measures appear in 


e 


- Table 1. One way analyses of variance were conducted’ on the posttest: 


means and are shown in Tables’? and: 3. - . é ° : 
,»4 . Insert Tables 1, 2, and 3 about here 


As/Gan be seen, the di¥ferences among group means obtained following 
. = ‘ t é . . 


treatment were reliable at Independent and Frustration Levels for the 
e Q 
oral reading ‘correct ‘measure, but not for the other dependent measures. / 
tf « 
Z . ° rf 
o\a post. hoc aS io dogag a Student- Newman-Keuls procedure was conducted | 


a 


' 


: and is presented in Table44. 


ty 


The results of ‘the paired comparisons revealed that the DMD group 
. ‘ ar , ” . Pa J? s ‘ 


oy performance exceeded the other three groups at both Frustration and. 
. @. . 

: % 

i} : e Ee 
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10 ‘ ge : | | : 
no - ? 


» 


Independent Levels with one exception? The -performanc 


was appayent.y egual to that . the DMN group, at, ave I 


ue 
ac 


e of the DMD group 


ndependent Level. 


~ 


The post hoc analysis also vegealed a difference ‘between DMN’ and ie UC 


° ° 
“4 . ve * 


“group at the Independent Level. 


whe < 


Discussion 


ue Bs 


~ 


FONG 


The results of the present study provide eVidence that variations 
* a 


in teacher measurement practi¢es and in how measuremen 


to make program decisions can significant ly influence 


t det8 are used 


° 7 - 


student performance. 


Severhl noteworthy conclusions concerning formative evaluation are sup" 
: . ‘ = ° 


¢ 


ponte by the obtained deta, 


“First, not ig ie ak alterations in formative 


evaluation pro- 


~ 


cedures, seem to impact most directly the behavior which is measured and. 


used we the a ei. tor ‘tnatyucf iofel! dt: making... 
: rr 


/ teachers | oral reading rate, used that data to 
* 


‘In this “research, 


make changes in 


the daily oral’ reading rate gears and’ ‘in whether or| what consequences 


were delivered for achieving those daily oral reading 
e 


goals. In addi- 


tion teachers recorded student performance on vocabulary- meaning, but 


bape 


set no daily objective for this. behavior and-did-not uge vocabulary 


- : * \ 


meaning data to make prograin changes. The results weré that treatment - 


effects were revealed in the oral reading correct data 


but not in the 


vocabulary meaning data nor in the standardized’ compréhension measure. 


The fact that treatment effects were not revealed in o 


- , » 


— 


ral reading errors 


weakens support for the conclusion that formative evaluation impacts mogt 


* 


directly che measured behavior. Given that error rate 


al] ane: reading levels, and that the daily aim fon e 


to maintain initial ‘error rates, iauousr, it is'npt su 


- 


iy¢ 


s ware tbe sieross 
rrors was deeeueially 


rprising that the 


4 


4 , . . 
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initial equivalence among treatment groups for this behavior remained at 
post test ing. The practical significance of this first conclusion rests 


aon he — that formative seatuskien Reg eS: that aseabe obtained on - 


* some particulat behaviors. If the advantage of perustive evaluation ry ye 
_ ‘ accrues prinarily to the measured behavior, then considerable importance 
myst be eacesiel in decisions regarding what behaviors to mepauke. : 
7 : A second conelusion pacing the pirecpice ucuponents of formative . ‘ 


evaluation which may be derived from the present results is that tradi- 


tional pre and posttesting on a particular objective does not contribute 
to improved achievement, Students whose aaa a in oral reading was 


meagured initially and again at the end of treatment “increased no more 
. v 
than students in the: untPeated ne group. That finding is made all 


3 


the more remarkable by the eae that the students in the pre and post 


a 
: _test treatment eerie’. Systematically practiced ora] reading each day 


white the students ‘in the ‘untreated tontrol group did not.' The impor- 


° 
° » 


a. 
tance of: this” failure of pre gnd posttesting as an approach to formative 


sualuation is that it calls into.question she purpose of the most 
. - : . . \ 


pervasive informal approach used by teachers to monitor student achieve- 
. 7 * . Py . . . 


ment. a 


7 ; 
The third notable — ees by the Sa that 


yo measurement of behavior specified in an ecur will not con- 


ia raise " performance heyond no testing at.all, and will not im- 


prove format ive evaluation pracedures beyond what {s obtained by pre and : 


posttesting. Student performance in oral reading in the daily’ meastrement 


condition exceeded that of the untreated control group at:‘the indépendent.-°. | 


* * ? oo s * 
reading level but .not at’ the frustration reading level. In none of the 


u . . Pao 
; s 2 2 


tion of measurement data are’included jas part of ‘the formative evaluation 


the untreated contro] and the pre-posttest gra@up at both Frustration and 


2 ? 4 ; = ae 


t ’ . oy . . 
comparisons did the mean ora’ reading -performance of students measured 


daily exceed that of the pre-post ‘testing SEOs Since students in. t he 
daily measurement aca did eieive ertornance feedback” ‘through 

ee 7% ° 
daily ingpect ion of their graphs and goals this finding is SUEDEEINE -_™ 
and err with the results: ebraanee by other eaaemuehers (e. Pes, 


% aos é 
Jenkins, Mayhall, Peschka, & omneond: 1974), One may speculate that 


the difference between their results and those “reported here may be - 
due to the differences in procedures and emphasis given to the daily 


feedback on the part of both student and teacher. The failure of daily | 
. ® uf al : ;” on 2 


measurement and chart ing to consistently increase student aghievement 
is impgrtant on effitjency grounds. ’ naily measurement and canting 
2 


is time consuming. If datty measurement is to be included as a component 


e 


of formative evaluation, ‘its role.in increasing student performance : 


° 


ra . 
i] 2 . 


should be demonstrated. ' 
_ ’ The most important conclusion which may be supported by the results 


of a present study is that systematic formative evaluation most “ef- | 


fectively contributes to apudene: ‘achievement when rules for. . the sutiliza- 


f 


system. When teachers measured student oral reading per farmance daily 
. y F 
in relation to daily goals and altered both goals and congequences con- 


‘ 


tingent upon measured student performance relative to goals, superior 


achievement occurred. It should be recalled that Students in the daily 


measurement treatment received exactly the same number and _type of pro- 


tae 


gram changes, on the same schedule, as ot ddeste in the dath-utilization 


+ a 


treatment, yet, in only one case did their performance exceed even the un- 
oy, i ‘ oe 


treated control group. In contrast, the data-utilization group exceeded 


\ 


* 1 p+ 
. ow 


a A 
\ ‘ ; ‘3 
&, . . 
. , 13 
, aS 
Independent. reading levels, and exceeded the daily measurement group 
é . 1 
at Frustration Leyel~ » 
Bo a Zi 


. We'need to taRe note that the data utilization treatment was a 


@ complex treatment. The separate effects of daily measurement, and 
be ; . , : ° . : . 
F rules for altering goals and delivering consequences cannot be determined. 


oe Teachers may be able to efficiently ‘alter goals and deliver consequences 
» 2 . e ; Ly 
' if they are not required to use. the particular data-utilizarfon rules 


. eupieyed in the present study, or if-they are not required to measure 


4 e 


daily. Our, position is, ‘however, that implicit in formative evaluation 
. - : > . a : 
» . iS a determinate.relation between measuremertt’ data and program changes, 
7 : ; 
» . at ‘ : ey 
and that formative evaluation consistently improves as improvements are 
¢ : | & 


£ 


€ . : i. : : : 
made in measurement and the procedures for utilizfng measurement data. 


The present results, we beliave, support the conclusion that daily measure 
: wy . t rs : 7 


Y : ‘ 


7 . a sake 2 o he ° . 
ment of student performanee is an important fomponent of formative eval- 


uation only when précedures for ytilizing daily performance data are re- 


. . . are af 


quired. 
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A final comment regarding measuring student performance in reading 


- for purposes of. formative evaluation should be made. One may argue 


rate in the basal reader is of. questionable importance 


that oral reading 
* . @, , ‘ : 
as an educational objective. -Fvidence is available to the contrary, a 


however. Oral reading “as a measure of ‘decoding skill is highly related - 


to reading achievement and to comprehension (Deno, Chiang, Mirkin, & 


, ° « Lowry, Note 9): pral reading performance, then, serves as a convenient 
7 Le } ' : * . 
e index of reading proficiency. An, interesting finding inthe present 
; x2 
study fs that oral reading at, Independent and Frustration Levels was 
a more sensitive to treatment effects. Jt may be that formative evaluatfon 
of reading requires regjildr measurement on content: externpl to dafly 
_— Iw 
ERIC _ a | 
4 » 
1 rent Provided by | 


t ee 
14 , 


a 


instruction. ue 


Further research on what to measure as a part of 
formative evaluation in reading is required 


e 
t 


The conclusions presented here are ies ea of course, by the 
' icy of the sa The only content iact was reading, and only a = 
. “eo 4 
' imited set of variations ‘pS measurement and data SEtiipaRion procedures, ‘ 
’ was employed... Such is the nature of an experiment, “ang we would prefer 
a ; ; nm 
an t : wee 4 ‘ ‘ 
to consider each of bur conclusions as hypotheses requiring extensive : 
2 * , Ps . ‘. 
testing with different subject matters, students, teachers, ‘tand, especially, ‘| 
2 a a a : + as . r 
with variations in formative évaluatign, procedures. 1 ¢ ; va 
oo . . 
| 
\ 
; , 
'y 
é ; : 
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Lohe oral reading rates used to gmitially Place. students torres-~ 


. bd ’ e 
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pond to. the lower end of the- independent, instruction ane frustration 


level wanes _Fegpmmboded by - Starlin (1973) when mak ing placement deci~. 


sions for actuary intermediate grade remedial students. ee 
‘ ; » » \ 


N . 
Since the data decision rule- ‘treatment requires a "program change. a 
, - : .. . 
whenever a student’ does fot: achieve the daily objective for -two consge- 
% r] ah te i af ' . ’ . 


cutive’ data days, program changes were also implemented for ‘students 


in the other experimental groups to Gontrol for the possibility that’ 


differences in performance could be attributed to the program changes 
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rather than to the formative evaluation procedures. 
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